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General  Problem  Statement 

>  DoD  Weapon  System  Failures  are  Attributed  to  Premature  Transfer 
of  Technology 

>  Current  DoD  Methods  to  Determine  Technology  Readiness  are 
I nadequate 

>  I  nsufficient  Measures  to  Assess  Technology  Readiness 

>  Lack  of  Quantifiable  and  Comparable  Risk  Assessments 

>  Maturity,  by  Itself,  is  Inadequate  to  Determine  Transition  Readiness 

>  Unmet  Performance  is  I  nsufficient  Measure  of  Risk 
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I  Why  is  this  a  Problem  for  DoD? 

>  Environment:  Technology  Development  Separated  from  Weapon 
System  Development 

>  Different  Priorities 

>  Different  Perspectives 

>  I  nconsi stent  Application  of  TRLs 

>  Unique  T echnology  Assessments 

>  Qualitative 

>  I  immature  Transition  of  Technology  Leads  to  Significant  Cost  and 
Schedule  I  impacts  Upon  DoD  Weapon  Systems 
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What  We  Know  &  Need  to  Know 

>  At  TRL  3,  Technologists  Know: 

Who . Customers 


What . Requi  rements 

When . Schedule 

How . Program  Plan 


>  I  n  Addition,  Technologists  NEED  to  Know: 

How  Well . PERFORMANCE  &  Rl  SK 

>  While  a  Thermostat  Measures  Actual  Temperature, 
the  Wind  Chill  Factor  is  More  of  Concern 


The  Proposed  Methodology  for  Calculating  Technology 
Performance  Risk  Measure  Provides  More  Realistic  Measure  of 
the  Actual  Performance  Risk,  J  ust  as  the  Wind  Chill  Factor 
Does  for  Temperature. 


Technology  Performance  Risk  Measure 


Garvey,  Paul  R.  and  Cho,  Chien-Ching,  “An  Index  to  Measure  a  System’s  Performance  Risk”, 

Acquisition  Review  Quarterly,  Spring  2003. 
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Specific  Problem  Statement 

>  Maturity,  as  a  One- Dimensional  Metric,  is  NOT  Adequate  to 
Determine  Technology  Transition  Readiness 

>  Unmet  Performance  is  NOT  Adequate  Risk  Measure 

>  The  Amount  of  Risk  Associated  with  Each  Requirement,  or  a 
Composite  Risk  at  the  System  Level,  Must  Be  I  ncorporated  to 
Provide  Realistic  and  Quantitative  Risk  Measure. 


Technology  Performance  Risk  Measure 


•egree  of  Difficulty 


IESCRIPTION 


Very  low  degree  of  difficulty  anticipated  in 
achieving  research  and  development  objectives 
for  this  technology;  only  a  single,  short-duration 
technological  approach  needed  to  be  assured  of  a  high 
probability  of  success  in  achieving  technical 
objectives  in  later  systems  applications 


Mankins,  1998  &  2002 


0.1 


Moderate  degree  of  difficulty  anticipated  in 
achieving  R&D  objectives  for  this  technology;  a 

single  technological  approach  needed;  conducted  early 
to  allow  an  alternate  approach  to  be  pursued  to  be  assured 
of  a  high  probability  of  success  in  achieving  technical 
objectives  in  later  systems  applications 


High  degree  of  difficulty  anticipated  in  achieving  R&D 
objectives  for  this  technology;  may  need  two  technological 
approaches;  conducted  early  to  allow  an  alternate  subsystem  approach 
to  be  pursued  to  be  assured  of  a  high  probability  of  success  in 
achieving  technical  objectives  in  later  systems  applications 


0.5 


IV 


V 


Very  high  degree  of  difficulty  anticipated  in  achieving  R&D 
objectives  for  this  technology;  risk  so  high  that  multiple  technological 
approaches  needed;  conducted  early  to  allow  an  alternate  system  concept  to  be 
pursued  to  be  assured  of  a  high  probability  of  success  in  achieving 
technical  objectives  in  later  systems  applications 

The  degree  of  difficulty  anticipated  in  achieving  R&D  objectives  for 
this  technology  is  so  high  that  a  fundamental  breakthrough  in  physics/ 
chemistry/etc.  is  needed;  basic  research  in  key  areas  needed  before 
feasible  system  concepts  can  be  refined 


0.9 
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Hypothetical  Example 


Ach 


Performance 


eved 


DD  =  0.1  DD  =  0 

DD  =  0.7  I  DD  =  0.9 


s'V  ,3  x 

^  ^ 

aOv1  aO^  aO^ 


■Threshold 


Degree  of 
Difficulty 

Risk  Level 

DD 

Value 

Theoretical 

No  Risk;  Guaranteed 

0.0 

Lower  Bound 

Success 

Level  1 

Very  Low  Risk 

0.1 

Level  2 

Low  Risk 

0.3 

Level  3 

Medium  Risk 

0.5 

Level  4 

High  Risk 

0.7 

Very  High  Risk, 

Level  5 

Requiring  Fundamental 

0.9 

Breakthrough 

Theoretical 
Upper  Bound 

Guaranteed  Failure 

1.0 
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I  Key  Assumptions 

>  Acquisition! st  and  Technologist  Reach  Agreement  Regarding 
Technical  Performance  Measures 

>  Technical  Performance  Measures  Provide  Sufficient  Quality- Level 
Requirements  to  Measure  Progress 

>  The  Degree  of  Difficulty  Numerical  Assignments  Provide 

Sufficient  Measure  and  Distinction  of  Performance  Risk  As  the 
TPM  Threshold  is  Achieved,  the  Technology  has  Moved  into  the 
Acceptable  Performance  Region 

>  As  the  TPM  Threshold  is  Achieved,  the  Technology  has  Moved 
into  the  Acceptable  Performance  Region 
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Technology  Performance  Risk  Measure 

Methodology 


Measured  Performance 

-  Performance  Metric 

-  Time 


Degree  of  Difficulty 

Risk  in  Unmet  Performance 
0  <  DD  <  1 


Threshold 


Achieved  Performance 

i 

Normalized 

Technology  Performance 
— ►  Risk  Measure 


Threshold 


o 


o 


TPRM  Decision  Aide 
Risk  Matrix 


o 


Requirement 
-  Threshold 


Requirement 

Criticality 
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System  Block  Diagram 


System 

Inputs 


DD 


Prioritization 
- ► 


TPRM 


System 

Output 


TPRM  =  li  m{F(P,DD)}  =  (1  -P) 


DD— >0 


TPRM  =  \[m{F(P;DD)}  =  0 


D— >1 
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Technology  Performance  Risk  Measure 

Mathematical  Model 

System  F(.) 


System  Inputs 


DD 


DD 


System  Output 

TPRM 


Y  =  V*  P 

V  =  X  +  Y*  DD 

X  =  (1-P)/P 


By  substituting  for  V, 

Y  =  (X  +  Y*DD)  *  P 

y  =  [(1-P)/P]*P  +  Y  *DD*P 
Y(1-DD*P)  =  [(1-P)/P  ]*P 

Y  =  (1-P)/(1-P*DD)  =  TPRM 
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Technology  Performance  Risk  Measure 

Mathematical  Model 


1=^ 


TPRM  =  (1-P)  /  (1-P*DD) 


Boundary  Conditions 

TPRM  =  lim  {F(P;DD)}  =  1 

DD  ->  1 

TPRM  =  lim  {F  (P;DD  )}  =  (1-P) 

DD  — »  0 


C  0.7 

o 

'ct> 

CD  0.6 
O' 

(/)  0.5 


DD  =  1  boundary 


Risk  Region 


TPRM  =  \[m  {F  (P;  DD  )}  =  1 

P^O 

TPRM  =  lim  {F(P-DD  )}  =  0 

p  ->  i 


0.2  0.3  0.4  0.5  0.6  0.7 

Achieved  Performance 
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echnology  Performance  Risk  Measure 

egree  of  Difficulty  as  Parameter 


TPRM  =  (l-P)  /  (1-P*DD) 


0.3  0.4  0.5  0.6  0.7 

Achieved  Performance 
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TPRM  Risk  Regions 


TPRM  =  (1-P)  /  (1-P*DD) 


Te 


■wyj  ■ 


IUI  IV^  I1IJI\ 
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Hypothetical  Example 


Performance 
Achieved 


DD  =  0.1  DD  =  0 

DD  =  0.7  I  DD  =  0.9 


Threshold 


\ 

^  ^  ^ 

-.<ev 

<SS>X  r\\>v  cS^V 


Requirement  1 
Requirement  2 
Requirement  3 
Requirement  4 


DD 


0.7 

0.1 

0.9 

N/A 


TPRM 


.69 
.43 
0.81 
0.00 


Aggregated  TPRM  =  0.48 
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Modified  Degree  of  Difficulty 


TPRM  Decision  Aide  Risk  Matrix 

Hypothetical  Example 


Technology  Performance  Risk  Measure 


TPRM  Success  Criterion 


TPRM  Transition 
Recommendation 


Program  Impact 


TPRM 

Successful? 


Yes 


No 


Greater  than  15%  cost  over  run  and/or 
greater  than  6  months  schedule  delay 

Greater  than  15%  cost  over  run  and/or 
greater  than  6  months  schedule  delay 
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Technology  Performance  Risk  Measure 

System  N:  System  Architecture 


m 


A1 

B1 

Cl 

A2 

B2 

C2 

A3 

B3 

C3 

A4 

B4 

C4 

System  N 

Requirement 

Level 


Sub-System 
Requirement 
Level  (3) 


Individual 
Requirement 
Level  (12) 
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Technology  Performance  Risk  Measure 

Case  Study  -  System  N 


Requirements 

Achieved 

Performance 

Degree  of 
Difficulty 

Individual 

Requirement 

TPRM 

Rank  Reciprocal 
Weights 

Subsystem  Level 
TPRM 

A1 

0.48 

0.70 

0.79 

0.08 

A 

A2 

0.86 

0.90 

0.62 

0.06  ; 

0.12 

A3 

1.00 

0.00 

0.00 

0.04 

A4 

0.39 

0.70 

0.84 

0.05 

B1 

1.00 

0.00 

0.00 

0.32  : 

B 

B2 

1.00 

0.00 

0.00 

0.16 

0.00 

B3 

1.00 

0.00 

0.00 

0.05 

B4 

1.00 

0.00 

0.00 

0.11 

Cl 

1.00 

0.00 

0.00 

0.03 

C 

C2 

0.46 

0.90 

0.92 

0.04 

0.05 

C3 

0.98 

0.70 

0.06 

0.03 

C4 

0.63 

0.70 

0.67 

0.03 

TPRM  (System  N)  =  0.17 
Government  Decision:  Continue  Development  but  do  not  Transition. 

TPRM  Agrees  with  Government  Decision  to  Continue  Technology 
Development.  Do  not  Transition  Since  Risk  for  Individual  Requirements 
within  Sub-Systems  A  and  C  Remains  High 
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TPRM  Decision  Aide  Risk  Matrix 

Case  Study  -  System  N 


Reqi 

TPRI 

liremer 
vi  =  0.6 

it  A2 

2 

MSm 

f2 

V ery  High 

Risk 

Tec 

TPI 

:hnolog 
IM  =  0. 

/  C4 

57 

.  High 

Technc 

TPRM 

fiisk 

•logy 
=  0.84 

u 

Requirement  C 
TPRM  =  0.06 

m 

• 

Reqi 

liremen 

t  A1 

Mediun 

iRilPRI 

1  =  0.7' 

Low 

Risk 

0.9  0.8  0.7  0.6  0.5  0.4  0.3 

Achieved  Performance 


0.2  0.1  0 
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I  Technology  Performance  Risk  Measure 

System  Q:  System  Architecture 

Evaluation  of  2  Technologies 
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Technical  Performance  Risk 
Measure 


■ 
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System  Q:  Technology  Comparisons 


Technology  1 

Requirement 

Performance 

Achieved 

Unmet 
Performance 
RISK  INDEX 

Degree 

of 

Difficulty 

TPRM 

Individual 

Requirement 

Rank 

Reciprocal 

Weight 

TPRM 

Sub-System 

Level 

A1 

A2 

A3 

A4 

0.49 

0.49 

0.24 

0.24 

0.51 

0.51 

0.76 

0.76 

0.5 

0.5 

0.7 

0.7 

0.68 

0.68 

0.91 

0.91 

1.72 

1.97 

1.83 

2.12 

0.80 

A1 

0.29 

0.71 

0.9 

0.96 

1.72 

Technology  2 

A2 

1 

0 

0 

0.00 

1.97 

A3 

1 

0 

0 

0.00 

1.83 

A4 

0.16 

0.84 

0.9 

0.98 

2.12 

0.49 

Modified  Degree  of  Difficulty 


TPRM  Decision  Aide  Risk  Matrix 

System  Q 
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Technology  Performance  Risk  Measure 

Summary 

S  Requisite  Decision  Model 

S  Easy  to  Utilize  and  Understand 

S  TPRM  Case  Studies  I  ndicated  Significant  Enhancement  in 
Determination  of  Technology  Transition  Readiness  Level 

S  Flexibility  to  Apply  to  Each  Level  of  Technology 

>  I  ndividual  Requirement  Level 

>  Sub- System  Category  Level 

>  Total  Technology  System  Level 

S  Provides  a  Quantitative- based  Assessment 

S  Value-Added  I  nformation  Regarding  Performance  Risk  to 
Support  Technology  Assessment  Related  Decisions 

S  Supports  Monitoring  of  Risks  Over  Time 

S  Supports  Prioritization  of  Resources  to  Mitigate  I  dentified  Risks 
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